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APPARATUS AND METHOD FOR STEREO MATCHING AND METHOD OF 
CALCULATING AN INFINITE DISTANCE CORRESPONDING POINT 

BACKGROUND OF THE INVENTION 

1 . Field of the Invention 

The present invention relates to an apparatus and a method 
for stereo matching and a method of calculating an infinite 
distance corresponding point . More particularly, the present 
invention relates to setting of a search range to be used in 
stereo matching. 

2. Description of the Related Art 

A stereo processing apparatus is known in the prior art, 
which obtains a pair of photographed images with a stereo camera 
system and calculates a distance to an object (or a parallax) 
shown in the photographed images based on the pair of 
photographed images obtained froma stereo camera system. First, 
the processing apparatus identifies a correlated destination 
of a given pixel block of one photographed image (right image, 
for example) in the other photographed image (left image, for 
example) (stereo matching) . Then, the processing apparatus 
calculates a parallax of the pixel block, that is, the amount 
of deviation in a horizontal direction between the position 
of the pixel block (a correlated source) on the right image 
and the position of the correlated destination on the left image . 



As is commonly known, the correlated destination of a given 
pixel block in the right image (reference image) exists on an 
epipolar line in the left image (comparative image) , or on the 
same horizontal line (at the same vertical coordinate) as the 
correlated source of the reference image is located, according 
to the principle of a stereo method. Accordingly, the parallax 
is calculated as the amount of rightward deviation with respect 
to a horizontal coordinate of the correlated source on the 
reference image. Here, assuming that the distance from the 
stereo camera system to an object is infinitely large, its 
theoretical parallax is zero and the correlated destination 
exists at the same horizontal coordinate as the correlated source . 
A point of the horizontal coordinate on the epipolar line of 
the correlated destination calculated when the obj ect is located 
at an infinite distance is referred to as an "infinite distance 
corresponding point." The amount of the rightward deviation 
(parallax) with respect to this infinite distance corresponding 
point increases as the distance to the object decreases. 
Therefore, a search range to be used when performing stereo 
matching may be set to be a predetermined range located to the 
right with respect to a point where the correlated destination 
is locatedmost leftward, or the infinite distance corresponding 
point . 

A coordinate point of the cor r el at ed source on the reference 
image (hereinafter referred to as a "reference coordinate 



point") and the infinite distance corresponding point on the 
comparative image theoretically coincide with each other as 
seen above. In actuality, however, there are many cases in 
which theydeviate from each other. The degree of such deviation 
is not uniform over the whole image but varies depending on 
the location in the image. A potential cause of this deviation 
is that focal distances of the right and left cameras differ 
from each other, causing one of the images to be enlarged or 
reduced compared to the other. The deviation can also occur 
due to distortion of camera lenses or an error in camera mounting 
position, for instance. As a consequence, there can arise such 
a situation that the infinite distance corresponding point is 
located to the left (which is opposite to the normal direction) 
of the reference coordinate point . Thus, there arises a problem 
that, if the reference coordinate point is set to as starting 
point of a search range, it becomes impossible to identify a 
correlated destination located to the left of this starting 
point . 

A potential solution to this problem would be to set the 
starting point of the search range to the left of the reference 
coordinate point with some margin. The amount of deviation 
of the infinite distance corresponding point from the reference 
coordinate point, however, is not uniform for different image 
areas as stated above. Therefore, when setting a margin of 
a fixed amount, it needs to be set to a larger amount than the 



amount of a maximum deviation of the infinite distance 
corresponding point. As a consequence, however, the search 
range widens as much as the increased amount of the margin thus 
provided, and this causes a problem that computational 
5 complexity involved in stereo matching increases, resulting 
in a reduction in processing speed. In addition, the 
possibility of mismatching increases as much as the increase 
in the search range . Consequently, there also arises a problem 
that the reliability of calculated parallaxes, or calculated 
10 distances, deteriorates. 

SUMMARY OF THE INVENTION 

The present invention has been made in view of the 
aforementioned problems. Accordingly, a first object of the 
present invention is to provide an apparatus and a method for 
15 stereo matching which can variably set a search range when 
performing stereo matching depending on the location on an image, 
and a method of calculating an infinite distance corresponding 
point . 

A second object of the present invention is to avoid an 
20 expansion of a search range by setting an appropriate search 
range depending on the location on an image, and thereby avoid 
an increase in computational complexity required for the stereo 
matching operation. 

A third object of the present invention is to reduce the 
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possibility of mi smatching by avoiding an expansion of the search 
range, and thereby prevent deterioration of the reliability 
of distance data. 

The obj ects can be achieved by a stereo matching apparatus 
which comprises a stereo camera system, a memory, an address 
generator, and a stereo matching unit . The stereo camera system 
takes pictures of a predetermined area and producing first and 
second images of the predetermined area. The memory stores 
first image data within a reference pixel region in the first 
image and second image data on a horizontal line in the second 
image, the horizontal line being corresponding to a vertical 
position of the reference pixel region in the first image. The 
address generator sets a search range for a stereo matching 
and instructs to read out form the memory a part of the second 
image data which is within the search range and the first image 
data within the reference pixel region. The stereo matching 
unit identifies a correlated destination of the reference pixel 
region by the stereo matching based on the part of the second 
image data which is within the search range and the first image 
data within the reference pixel region which are read out from 
the memory and calculates a parallax of the reference pixel 
region based on the correlated destination of the reference 
pixel region. The above-mentioned address generator corrects 
a location of the search range for the reference pixel region 
based on the amount of deviation of an infinite distance 



corresponding point with respect to a horizontal position of 
the reference pixel region. 

In the stereo matching apparatus of the invention, it is 
preferable that the address generator shifts the location of 
the search range in the same direction as a direction of the 
deviation of the infinite distance corresponding point with 
respect to the horizontal position of the reference pixel region . 
The address generator may shift a starting point of the search 
range in the same direction as the direction of the deviation 
of the infinite distance corresponding point. Further, the 
address generator may shift an end point of the search range 
in the same direction as the direction of the deviation of the 
infinite distance corresponding point. 

Further, in the stereo mat ching apparatus of the invention, 
it is preferable that the address generator identifies the amount 
of correction of the search range in relation to the amount 
of deviation of the infinite distance corresponding point. 

Furthermore, it is also preferable that the stereo matching 
apparatus further comprises a correction value memory. The 
correction valuememory stores a correction value for correcting 
the location of the search range in relation to a location of 
the reference pixel region in the first image. In this case, 
the -address generator sets the location of the search range 
for the reference pixel region based on the location of the 
reference pixel region and the correction value stored in the 



correction value memory for the location of the reference pixel 
region. The address generator may set the starting point of 
the search range for the reference pixel region based on the 
location of the reference pixel region and the correction value 
5 stored in the correction value memory for the location of the 
reference pixel region. 

In the above-mentioned stereo matching apparatus, the 
stereo camera system may comprise a pair of stereo cameras 
mounted on a vehicle, the pair of stereo cameras taking pictures 
10 of scenes outside of the vehicle. In this case, the stereo 
matching unit calculates a distance to an object outside the 
vehicle on the basis of the parallax of the reference pixel 
region . 

The above-mentioned objects can be also achievedby a stereo 
15 matching method comprising: taking pictures of a predetermined 
area and producing first and second images of the predetermined 
area; identifying a reference pixel region in the first image; 
identifying a search range for the reference pixel region in 
the second image; correcting a location of the search range 
20 for the reference pixel region based on the amount of deviation 
of an infinite distance corresponding point with respect to 
a horizontal position of the reference pixel region; and 
identifying a correlated destination of the reference pixel 
region by searching through the search range. 
25 In the stereo matching method of the invention, it is 
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preferably advantageous that the step of correcting the location 
of the search range shifts the location of the search range 
in the same direction as the direction of the deviation of the 
infinite distance corresponding point with respect to the 
5 horizontal position of the reference pixel region. The step 
of correcting the location of the search range may shift a 
starting point of the search range in the same direction as 
the direction of the deviation of the infinite distance 
corresponding point with respect to the horizontal position 

10 of the reference pixel region. Further, the step of correcting 
the location of the search range may shift an end point of the 
search range in the same direction as the direction of the 
deviation of the infinite distance corresponding point with 
respect to the horizontal position of the reference pixel region. 

15 In the above-mentioned stereo mat ching method of invention, 

it is further advantageous that the step of correcting the 
location of the search range corrects the location of the search 
range with a correction value corresponding to the deviation 
of the infinite distance corresponding point. 

20 Further, it is advantageous that the stereo matching method 

further comprises: calculating a distance to an object outside 
a vehicle on the basis of the correlated destination of the 
reference pixel region. In this case, the step of taking 
pictures of the predetermined area takes pictures of scenes 

25 outside of the vehicle. 

8 



Furthermore, it is advantageous that the stereo matching 
method further comprises: taking picture of a screen by a pair 
of stereo cameras and producing third and fourth images of the 
screen, the screen being drawn a first vertical line and a second 
5 vertical line at an interval equal to a baseline of the stereo 
cameras; identifying a deviation of a position of the second 
vertical line shown on the fourth image with respect to a position 
of the first vertical line shown on the third image; and 
calculating an infinite distance corresponding point 

10 corresponding to each position on the first vertical line shown 
on the third image on the basis of the deviation of the position 
of the second vertical line . In this case, the step of correcting 
the location of the search range uses the infinite distance 
corresponding point thus calculated as the infinite distance 

15 corresponding point corresponding to the horizontal position 
of the reference pixel region. 

Further, the above-mentioned objects can be achieved a 
stereo matching apparatus which comprises a stereo camera system, 
a reference pixel region identifying unit, a search range 

20 identifying unit , a correction unit and a correlated destination 
identifying unit. The stereo camera system takes pictures of 
a predetermined area and producing first and second images of 
the predetermined area . The reference pixel region identifying 
unit identifies a reference pixel region in the first image. 

25 The search range identifying unit identifies a search range 



9 



for the reference pixel region in the second image. The 
correction unit corrects a location of the search range for 
the reference pixel region based on the amount of deviation 
of an infinite distance corresponding point with respect to 
5 a horizontal position of the reference pixel region. The 
correlated destination identifying unit identifies a 
correlated destination of the reference pixel region by 
searching through the search range. 

Furthermore, the above-mentioned objects can be achieved 

10 by a method of calculating an infinite distance corresponding 
point. The method comprising: taking picture of a screen by 
a pair of stereo cameras and producing first and second images 
of the screen, the screen being drawn a first vertical line 
and a second vertical line at an interval equal to a baseline 

15 of the stereo cameras; identifying a deviation of a position 
of the second vertical line shown on the second image with respect 
to a position of the first vertical line shown on the first 
image; and calculating an infinite distance correspondingpoint 
corresponding to each position on the first vertical line shown 

20 on the first image on the basis of the deviation of the position 
of the second vertical line. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a block diagram of a stereo processing apparatus 
according to an embodiment of the present invention; 
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Fig. 2 is a diagram showing a search range when a correction 
value is set to 0; 

Fig. 3 is a diagram showing a search range when a correction 
value is set to +4; 
5 Fig. 4 is a diagram showing a search range when a correction 

value is set to -4; 

Fig. 5 is a diagram for explaining a method of calculating 
an infinite distance corresponding point according to the 
present invention; 
10 Fig. 6 is a diagram showing ideal photographed images; 

Fig. 7 is a diagram showing photographed images obtained 
in a case where a focal distance of a left camera is smaller 
than a focal distance of a right camera; 

Fig. 8 is a diagram showing photographed images obtained 
15 in a case where a focal distance of the left camera is smaller 
than a focal distance of the right camera and a lens of the 
left camera is greatly distorted; and 

Fig . 9 is a diagram showing a relationshipbetween reference 
coordinate points and correction values 

2 0 DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

Fig. 1 is a block diagram of a stereo processing apparatus 
according to an embodiment of the present invention. A stereo 
camera system 1 which takes pictures of views outside a vehicle 
is mounted in the vicinity of its rear- view mirror . The stereo 
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camera system 1 is composed of a pair of cameras 2a, 2b, each 
incorporating an image sensor such as a charge-coupled device 
(CCD) . These cameras 2a, 2b are attached with a predetermined 
spacing in the direction of vehicle width therebetween. The 
5 main camera 2a for obtaining a reference image data is attached 
on the right side of the vehicle's forward direction. On the 
other hand, the sub-camera 2b for obtaining a comparative image 
is attached on the left side of the forward direction. 

In a condition in which the pair of the cameras 2a, 2b 

10 is synchronized, analog images output from the cameras 2a, 2b 
are respectively converted by analog-to-digital (A/D) 
converters 3a, 3b into digital images , each digital image having 
a predetermined brightness level (e.g., 256 gray scale levels ) 
As a consequence, reference image data (brightness data) of 

15 512 pixels in the horizontal direction by 200 pixels in the 
vertical direction is produced from an output signal of the 
main camera 2a. Also, from an output signal of the sub-camera 
2b, comparative image data is obtained, the comparative image 
having the same vertical length as the reference image and a 

20 greater horizontal length than the reference image (640 pixels 
in the horizontal direction by 200 pixels in the vertical 
direction, for example) . The digitized stereoscopic images 
(reference image and comparative image) for one frame (one view) 
are individually stored in frame memories 4a, 4b having a 

25 capacity to store digital data for one frame. 
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The data stored in predetermined memory areas in the frame 
memories 4a, 4b are read out according to instructions from 
an address generator 12 which constitutes part of an address 
generating circuit 10. The image data thus read out are 
transferred to an image correction circuit 5. The image 
correction circuit 5 carries out such operations as a position 
correction and a brightness correction of the images so as to 
increase the accuracy of stereo matching which is performed 
by a stereo matching circuit 8 in a later stage. There are 
cases where the positions of the stereoscopic images are 
displaced due to a deviation of mounting positions (in a 
horizontal, vertical or rotational direction) of the right and 
left cameras 2a, 2b, a distortion of camera lenses, or an 
inclination of a light-sensing surface of each CCD, for instance . 
Accordingly, the deviation o f the stereoscopic images is reduced 
by applying a linear shape correct ion like af f ine transformation 
or a nonlinear shape correction to the stereoscopic images. 
A nonlinear correction method that can be used, for example, 
is a method described in Japanese Patent Application No. Hei . 

10- 132275 (Japanese Patent Unexamined Publication No. Hei. 

11- 325889) . The brightness correction is a correction for 
correcting differences in the sensitivity of the stereo 
processing apparatus . The brightness correction, for example, 
is a LOG correction for improving the contrast of low-brightness 
parts (described in Japanese Patent Application No. Hei. 



11-6879 ( Japanese Patent Unexamined Publication No. 
2000-207555) ) , a shading correction for correcting brightness 
values based on a shading correction ratio stored for each pixel 
(described in Japanese Patent Application No. Hei . 
5 5-58974 ( Japanese Patent Unexamined Publication No. Hei. 
06-273171)), etc. 

The reference image data corrected by the image correction 
circuit 5 is stored in a memory area in a reference image line 
memory 6. More specifically, the reference image data of four 

10 horizontal lines in pixel units (one horizontal line in pixel 
block units 4x4) is stored. Also, the comparative image data 
corrected by the image correction circuit 5 is stored in a memory 
area in a comparative image line memory 7. More specifically, 
the comparative image data of one horizontal line in pixel block 

15 units is stored. The expression "one horizontal line" used 
in the following explanation refers to one horizontal line in 
pixel block units. The address generator 12 specifies the 
memory areas of the line memories 6, 7 in which the image data 
are written. The reference image data output from the image 

20 correction circuit 5 is also entered to a microcomputer 9. 

The reference image data corresponding to one pixel 
block (reference pixel block) stored in the reference image line 
memory 6 is read out according to an instruction from the address 
generator 12 and transferred to the stereo matching circuit 

25 8. The data thus transferred is correlated source data to be 
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used when per forming stereo matching operation . In other words , 
it is pixel data of eachpixel (brightness value ) in the reference 
pixel block (hereafter, referred to as "reference pixel block 
data") . Also, the comparative image data corresponding to a 
"search range" stored in the comparative image line memory 7 
(hereafter referred to as "search range data") is read out 
according to an instruction from a search range address generator 
13 and transferred to the stereo matching circuit 8. The data 
thus transferred is pixel data of each pixel (brightness value) 
to be used in the stereo matching (hereafter , referred to as 
"search range data") . As the "search range" set by the search 
range address generator 13 is one of features of the present 
embodiment, it will be described later in detail. 

The stereo matching circuit 8 calculates parallaxes by 
performing the stereo matching based on the reference pixel 
block data and the search range data. Since the parallax is 
calculated for each pixel block of 4x4 pixels, a maximum of 
128x50 parallaxes can be obtained from one complete frame of 
the reference image. Considering one pixel block (reference 
pixel block) in the reference image, a region (correlated 
destination) having a correlation with brightness properties 
of the one pixel block is identified by searching through the 
search range in the comparative image. While shifting a search 
point pixel by pixel along the aforementioned epipolar line, 
the stereo matching circuit 8 evaluates the state of correlation 



of each pixel block within the search range with the reference 
pixel block. 

The correlation between two pixel blocks can be evaluated 
by calculating city block distance CB shown in the following 
equation (1), for instance. In this equation, plij represents 
a brightness value of pixel (i, j) of one pixel block, while 
p2ij represents a brightness value of pixel (i, j ) of the other 
pixel block. The city block distance CB is the sum (absolute 
value) of differences between the two geographically 
corresponding brightness values plij, p2ij for one complete 
pixel block, wherein the smaller the differences, the greater 
the correlation between the two pixel blocks. 

CB = Zlplij - p2ij | (1) 

Basically, a pixel block which gives a minimum value of 
city block distance CB among city block distances CB calculated 
for the each pixel block existing on the epipolar line is judged 
to be the correlated destination of the reference pixel block. 
The amount of deviation between the correlated destination thus 
identified and the reference pixel block is the parallax. 

The stereo matching circuit 8 also evaluates the presence 
or absence of abrightness edge (the amount of brightness change) 
in the horizontal direction of the pixel block, as well as its 
relationship with maximum and minimum values of the city block 
distance CB calculated on the same epipolar line . Furthermore, 
to ensure reliability as distance data, the stereo matching 



circuit 8 performs a filtering operation on the parallax and 
outputs only such parallax that has been judged to be valid 
(the parallax which has been judged to be valid is hereinafter 
referred to simply as the "parallax") . Reference should be 
5 made to Japanese Patent Unexamined Publication No. 5-114099 
for details of hardware configuration and filtering operation 
for calculating the city block distance, if necessary, since 
they are disclosed therein. The parallax calculated through 
the aforementioned operation is transferred to the 

10 microcomputer 9. 

In this embodiment, the address generating circuit 10 has 
a function of setting a search range S to be used when performing 
the stereo matching. To realize this function, the address 
generating circuit 10 has a reference pixel block position 

15 designator 11 , a correction value memory 14 and the search range 
address generator 13. The position where the search range S 
is set is corrected as follows according to the status of an 
infinite distance corresponding point % of the position (ip, 
jp) of a reference pixel block PB . 

20 First, for the reference pixel block PB to be subjected 

to the stereo matching, the reference pixel block position 
designator 11 identifies its position (ip, jp) on the reference 
image (where 0<ip<127, 0<jp<49) in pixel block units. Then, 
the position (ip, jp) of the reference pixel block PB is output 

25 to the search range address generator 13 and the correction 
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value memory 14. 

When the position (ip, jp) of the reference pixel block 
PB has been identified by the reference pixel block position 
designator 11, reference is made to a search starting point 
5 correction table 15 stored in the correction value memory 14 , 
and then a correction value PS for the reference pixel block 
PB is identified. The correction value memory 14 is formed 
of an electrically erasable programmable read-only memory 
(EE PROM) , for example. This correction value PS is a value 

10 for correcting a starting point of the search range S in the 
comparative image, or for correcting the search starting point, 
applicable when searching for a correlated destination of the 
reference pixel block PB . Correction values PS are set and 
written in the correction value memory 14, based on deviation 

15 characteristics of an infinite distance corresponding point 
% on a photographed image measured at the time of a system 
initialization (i.e., in an adjustment process in a factory 
or during readjustment at a dealer) . The infinite distance 
corresponding point % and the correction value PS generally 

20 have the following relationship: 

(Relationship between infinite distance corresponding point 
X and correction value PS) 

* 1. The correction value PS is of the same sign as the 
direction (plus or minus) of deviation of the infinite distance 

25 corresponding point % with respect to a reference coordinate 
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point X (the coordinate point of the reference pixel block PB) . 

2. The greater the amount of deviation of the infinite 
distance corresponding point % with respect to the reference 
coordinate point X, the greater the amount of correction that 
the correction value PS indicates. 

When the correction value PS corresponding to the position 
(iP/ jp) of the reference pixel block has been read out from 
the correction value memory 14, a starting point SAD of a search 
range S and an end point EAD thereof are calculated based on 
the following equations by the search range address generator 
13 and the search range S is set: 

(Setting of search range S) 

Starting point SAD = ( 4 x ip - /3 ) + PS 

End point EAD = SAD + 131 

Specifically, the reference coordinate point X (in pixel 
units) is calculated by multiplying the horizontal coordinate 
ip of the reference pixel block PB by the number of horizontally 
arranged pixels 4 in one pixel block. A basic starting point 
A is calculated by subtracting a specified margin 0 from the 
reference coordinate point X (or by shifting the reference 
coordinate point X leftward) . The starting point SAD of the 
search range S is a value obtained by adding the correction 
value PS to the basic starting point A. Also, the end point 
EAD of the search range S is a value obtained by adding a specified 
value 131 to the starting point SAD, so that this value is uniquely 



determined when the starting point SAD is determined. The 
search range S is therefore the range (as wide as 132 pixels) 
from the starting point SAD to the end point EAD . 

The search range address generator 13 gives an instruction 
to the comparative image line memory 7 that the comparative 
image data (search range data) within the search range S thus 
set should be read out. The search range data read out is 
transferred to the stereo matching circuit 8. The stereo 
matching circuit 8 then calculates the aforementioned city block 
distance CB based on the reference pixel block data transferred 
from the reference image line memory 6 and the search range 
data . 

Fig. 2 is a diagram showing a search range S set when a 
correction value PS is 0. The correction value PS is set 
according to its relationship with the measured infinite 
distance corresponding point % as described above . Thus, the 
correction value PS of 0 means that the deviation of the infinite 
distance corresponding point % with respect to the reference 
coordinate point X is 0 or, in other words, the infinite distance 
corresponding point % exists at the same hori zontal coordinate 
as the reference coordinate point X. Therefore, even when 
search operation is begun from the basic starting point A 
(correction value PS = 0) , the infinite distance corresponding 
point % will in no case depart from the search range S. The 
end point EAD of the search range S in this case is A + 131. 
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Fig. 3 is a diagram showing a search range S set when a 
correction value PS is + 4 . Since the correction value PS is 
+ 4, the starting point SAD is set to a value obtained by adding 
the correction value 4 to the basic starting point A and the 
5 end point EAD is set to +135. The correction amount PS of +4 
indicates that the infinite distance corresponding point % 
deviates rightward by the amount of correction equal to 4 with 
respect to the reference coordinate point X. Therefore, even 
when the starting point SAD is shifted rightward by the amount 

10 of correction, the infinite distance corresponding point % 
will in no case depart from the search range S, and it is assured 
that the infinite distance corresponding point % exists in the 
right-hand vicinity of the starting point SAD. 

Fig. 4 is a diagram showing a search range S set when a 

15 correction value PS is -4. Since the correction value PS is 
-4, the starting point SAD is set to a value obtained by 
subtracting the correction value 4 from the basic starting point 
A and the end point EAD is set to +127. The correction amount 
PS of -4 indicates that the infinite distance corresponding 

20 point x deviates leftward by the amount of correction equal 
to 4 with respect to the reference coordinate point X . Therefore , 
it is possible to prevent the infinite distance corresponding 
point x from departing from the search range S by shifting the 
starting point SAD leftward as much as the amount of correction . 

25 It is assured that the infinite distance corresponding point 
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X exists in the right-hand vicinity of the starting point SAD 
in this case as well. 

Considering that the deviation of the infinite distance 
corresponding point % with respect to the reference coordinate 
5 point X is not uniform for different image areas, the starting 
point SAD and the end point EAD of the search range S are shifted 
in this embodiment. In contrast, if the search range S is set 
at a fixed location, it is necessary to shift the starting point 
of the search range S leftward by a margin determined in 

10 consideration of the amount of a maximum deviation of the 
infinite distance corresponding point x and also shift its end 
point rightward by a relatively large margin. If the location 
of the search range is made variable according to the deviation 
of the infinite distance corresponding point as in the present 

15 embodiment, it is not necessary to set such large margins so 
that an expansion of the search range will not result. As a 
consequence, itispossibleto avoid an increase in computational 
complexity required for the stereo matching and to prevent a 
reduction in processing speed. It is also possible to reduce 

20 the possibility of mismatching in the stereo matching, because 
it is not necessary to set a large search range. It is therefore 
possible to ensure the accuracy of calculated parallaxes, or 
calculated distances. 

In the above-mentioned embodiments, the starting point 

25 SAD of the search range S is obtained by adding the correction 
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value PS to the basic starting point A, and the end point EAD 
thereof is obtained by adding the specified value 131 to the 
starting point SAD. However, the ending point EAD of the search 
range S may be obtained by adding the correction value PS. In 
5 this case, the starting point SAD is obtained by subtracting 
the specified value 131 to the ending point EAD. The deviation 
of the infinite distance corresponding point % with respect 
to the reference coordinate point X can be obtained by the 
following method. Fig. 5 is a diagram for explaining a method 

10 of calculating the infinite distance corresponding point % 
according to the present invention. First, to identify the 
infinite distance corresponding point % , a special screen 21 
is placed at a predetermined position in front of the stereo 
camera system 1 . Six vertical lines are drawn on this screen 

15 21, wherein an interval between the adjacent vertical lines 
is made equal to a length of the baseline of the stereo camera 
system 1, or to an interval between the right camera 2a and 
the left camera 2b. The individual vertical lines are numbered 
to permit their identification. Specifically, the rightmost 

20 vertical line on the screen is numbered "comparison No. 1," 
from where the rest of the vertical lines are sequentially 
numbered "comparison No . 2" to "comparison No . 5." Also, the 
second vertical line from the right side of the screen is numbered 
"reference No. 1," from where the rest of the vertical lines 

25 are sequentially numbered "reference No. 2" to "reference No. 
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5. " The screen 21 is positioned in such a way that the vertical 
line "reference No. 3" would be shown at the center of a 
photographed image (reference image) when the screen 21 is 
photographed by the right camera 2a. As a result, the vertical 
line "comparison No . 3" is shown at the center of a photographed 
image (comparative image) taken by the left camera 2b. 

When such screen 21 is photographed with the stereo camera 
system 1, photographed images as shown in Fig. 6 are obtained 
under an ideal condition. Under this condition, the positions 
of the vertical lines "comparison No. 1" to "comparison No. 
5" shown on the comparative image perfectly coincide with the 
positions of the corresponding vertical lines "reference No. 
1" to "reference No. 5" shown on the reference image. The 
interval between the individual vertical lines is made equal 
to the length of the baseline of the stereo camera system 1. 
Therefore, according to characteristics of the stereo method, 
the corresponding vertical lines "comparison No. 1" to 
"comparison No. 5" on the comparative image correspond to 
infinite distance corresponding points % at the respective 
positions (their deviation is 0 in this case) . 

If there is any influence of variations in focal distances 
or lens distortion of the right and left cameras 2a, 2b, or 
an error in horizontal positions of right and left cameras 2a, 
2b, the photographed images as shown in Fig. 6 would not be 
obtained. Fig. 7 is a diagram showing photographed images 
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obtained in a case where the focal distance of the left camera 
2b is smaller than the focal distance of the right camera 2a 
(where the left camera 2b is more wide-angled) . In this case, 
the comparative image shown will be reduced in the horizontal 
direction, centering on the center of the image. Therefore, 
the vertical lines "comparison No. 1" and "comparison No. 2" 
are shifted rightward compared to those under the ideal condition, 
while the vertical lines "comparison No. 4" and "comparison 
No. 5" are shifted leftward compared to those under the ideal 
condition. For this reason, an object (which has a finite 
distance) shown on the individual vertical lines "reference 
No. 1" to "reference No. 5" is always located to the right of 
the positions of their corresponding vertical lines "comparison 
No. 1" to "comparison No. 5" (i.e., the infinite distance 
corresponding points % ) . 

Fig, 8 is a diagram showing photographed images obtained 
in a case where the focal distance of the left camera 2b is 
smaller than the focal distance of the right camera 2a and the 
lens of the left camera 2b is greatly distorted. Since the 
aforementioned factors often exist in combination in actuality, 
there are many cases where the photographed images shown in 
Fig. 8 are obtained. Unlike the case of Fig. 7, the deviation 
of the corresponding vertical lines "comparison No. 1" to 
"comparison No. 5" varies depending not only on horizontal 
location but also on vertical location in the present case. 



Namely, the deviation of the infinite distance corresponding 
points % varies depending on the vertical location in the image 
areas . 

The deviation of the infinite distance corresponding point 
5 % for each location in the image areas is determined and the 
correction value PS corresponding to the deviation is 
individually set for each location as seen above. Fig. 9 is 
a diagram showing a relationship between reference coordinate 
points (coordinate points of reference pixel blocks) and 

10 correction values PS . When the infinite distance corresponding 
points % coincide with their corresponding reference 
coordinate points X over the whole of horizontal coordinates 
on the images as illustrated in Fig. 6, the correction value 
PS is set to 0 as shown by a symbol "a" in Fig. 9. In contrast, 

15 when the vertical lines "reference No. 1," "reference No. 2," 
"reference No. 4" and "reference No. 5" are shifted toward the 
central vertical line "reference No. 3" as illustrated in Fig. 
7, the correction value PS is set in such a way that it linearly 
decreases with an increase in horizontal coordinate value as 

20 shown by a symbol "b" in Fig . 9. Also , under conditions depicted 
in Fig. 8, correction values PS are set in such a way that it 
nonlinearly decrease with an increase in horizontal coordinate 
value as shown by a symbol "c" in Fig. 9. A curve "c" of Fig. 
9 shows the correction values PS for the same vertical coordinate . 

25 Furthermore, when there exists a deviation in a yaw direction 
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between the right and left cameras 2a , 2b, characteristics shown 
in Fig. 9 shift in a vertical direction. 

It is possible to determine the deviation of the infinite 
distance corresponding points % with respect to the reference 
5 coordinate points by measuring the deviation of corresponding 
vertical lines "comparison No. 1" to "comparison No. 5" to 
vertical lines "reference No. 1" to "reference No. 5" as seen 
above. As thus far described, the correlated destination of 
a given reference pixel block PB always exists to the right 

10 of the corresponding vertical line "comparison No. 1" which 
corresponds to the infinite distance corresponding point % . 
Therefore, the correction value PS should be set to a search 
point SAD which always includes this infinite distance 
corresponding point % . 

15 It is possible to set an appropriate search range for each 

reference pixel block by correcting the location of the search 
range according to the deviation of an infinite distance 
corresponding point with respect to a reference coordinate point 
of the reference pixel block as seen above in this invention. 

20 Therefore, it is not necessary to set a large margin for the 
search range considering a maximum deviation of the infinite 
distance corresponding point as described above. Since an 
expansion of the search range can be avoided as a consequence, 
it is possible to avoid an increase in computational complexity 

25 required for stereo matching. Furthermore, since the 
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possibility of mismatching can be reduced by avoiding the 
expansion of the search range, it is possible to prevent 
deterioration of the reliability of distance data. 

While the presently preferred embodiment of the present 
5 invention has been shown and described, it is to be understood 
that this disclosure is for the purpose of illustration and 
that various changes and modifications may be made without 
departing from the scope of the invention as set forth in the 
appended claims. 
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WHAT IS CLAIMED IS: 

1. A stereo matching apparatus comprising: 

a stereo camera system for taking pictures of a 
predetermined area and producing first and second images of 
5 the predetermined area; 

a memory for storing first image data within a reference 
pixel region in the first image and second image data on a 
horizontal line in the second image, the horizontal line being 
corresponding to a vertical position of the reference pixel 
10 region in the first image; 

an address generator for setting a search range for a stereo 
matching and instructing to read out from said memory a part 
of the second image data which is within the search range and 
the first image data within the reference pixel region; and 
15 a stereo matching unit for identifying a correlated 

destination of the reference pixel regionby the stereo matching 
based on the part of the second image data which is within the 
search range and the first image data within the reference pixel 
region which are read out from said memory and calculating a 
20 parallax of the reference pixel region based on the correlated 
destination of the reference pixel region, 

wherein said address generator corrects a location of the 
search range for the reference pixel region based on the amount 
of deviation of an infinite distance corresponding point with 
25 respect to a horizontal position of the reference pixel region. 
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2. The stereo matching apparatus as recited in claim 

1, wherein said address generator shifts the location of the 
search range in the same direction as a direction of the deviation 

5 of the infinite distance corresponding point with respect to 
the horizontal position of the reference pixel region. 

3. The stereo matching apparatus as recited in claim 

2, wherein said address generator shifts a starting point of 
10 the search range in the same direction as the direction of the 

deviation of the infinite distance corresponding point. 

4. The stereo matching apparatus as recited in claim 
2, wherein said address generator shifts an end point of the 

15 search range in the same direction as the direction of the 
deviation of the infinite distance corresponding point. 

5 - The stereo matching apparatus as recited in claim 

1, wherein said address generator identifies the amount of 
20 correction of the search range in relation to the amount of 

deviation of the infinite distance corresponding point. 

*6. The stereo matching apparatus as recited in claim 

2, wherein said address generator identifies the amount of 
25 correction of the search range in relation to the amount of 



30 



deviation of the infinite distance corresponding point. 



7. The stereo matching apparatus as recited in claim 

3, wherein said address generator identifies the amount of 
correction of the search range in relation to the amount of 
deviation of the infinite distance corresponding point. 

8 . The stereo matching apparatus as recited in claim 

4, wherein said address generator identifies the amount of 
correction of the search range in relation to the amount of 
deviation of the infinite distance corresponding point. 

9. The stereo matching apparatus as recited in claim 

2, further comprising: 

a correction value memory for storing a correction value 
for correcting the location of the search range in relation 
to a location of the reference pixel region in the first image; 

wherein said address generator sets the location of the 
search range for the reference pixel regionbased on the location 
of the reference pixel region and the correction value stored 
in said correction value memory for said location of the 
reference pixel region. 

10. The stereo matching apparatus as recited in claim 

3, further comprising: 
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a correction value memory for storing a correction value 
for correcting the location of the search range in relation 
to a location of the reference pixel region in the first image; 
wherein said address generator sets the starting point 
5 of the search range for the reference pixel region based on 
the location of the reference pixel region and the correction 
value stored in said correction value memory for said location 
of the reference pixel region. 



10 11. The stereo matching apparatus as recited in claim 

1, wherein said stereo camera system comprises a pair of stereo 
cameras mounted on a vehicle, the pair of stereo cameras taking 
pictures of scenes outside of the vehicle, and said stereo 
matching unit calculates a distance to an object outside the 

15 vehicle on the basis of the parallax of the reference pixel 
region . 



12. A stereo matching method comprising: 

taking pictures of a predetermined area and producing first 
20 and second images of the predetermined area; 

identifying a reference pixel region in the first image; 

identifying a search range for the reference pixel region 
in the second image; 

correcting a location of the search range for the reference 
25 pixel region based on the amount of deviation of an infinite 
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distance corresponding point with respect to a horizontal 
position of the reference pixel region; and 

identifying a correlated destination of the reference 
pixel region by searching through the search range. 

5 

13. The stereo matching method as recited in claim 12, 
wherein said step of correcting the location of the search range 
shifts the location of the search range in the same direction 
as the direction of the deviation of the infinite distance 

10 corresponding point with respect to the horizontal position 
of the reference pixel region. 

14. The stereo matching method as recited in claim 13, 
wherein said step of correcting the location of the search range 

15 shifts a starting point of the search range in the same direction 
as the direction of the deviation of the infinite distance 
corresponding point with respect to the horizontal position 
of the reference pixel region. 

20 15 - The stereo matching method as recited in claim 13, 

wherein said step of correcting the location of the search range 
shifts an end point of the search range in the same direction 
as the direction of the deviation of the infinite distance 
corresponding point with respect to the horizontal position 

25 of the reference pixel region. 
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16. The stereo matching method as recited in claim 12, 
wherein said step of correcting the location of the search range 
corrects the location of the search range with a correction 
value corresponding to the deviation of the infinite distance 
corresponding point. 



17 . The stereo matching method as recited in claim 13, 
wherein said step of correcting the location of the search range 
corrects the location of the search range with a correction 
value corresponding to the deviation of the infinite distance 
corresponding point. 



18. The stereomatchingmethod as recited in 14, wherein 
said step of correcting the location of the search range corrects 
the location of the search range with a correction value 
corresponding to the deviation of the infinite distance 
corresponding point. 



19 . The stereo matching method as recited in 15, wherein 
said step of correcting the location of the search range corrects 
the location of the search range with a correction value 
corresponding to the deviation of the infinite distance 
corresponding point. 
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20 . The stereo matching method as recited in claim 12, 
further comprising: 

calculating a distance to an object outside a vehicle 
on the basis of the correlated destination of the reference 
5 pixel region, 

wherein said step of taking pictures of the predetermined 
area takes pictures of scenes outside of the vehicle. 

21. The stereo matching method as recited in claim 12, 
10 further comprising: 

taking picture of a screen by a pair of stereo cameras 
and producing third and fourth images of the screen, said screen 
being drawn a first vertical line and a second vertical line 
at an interval equal to a baseline of the stereo cameras; 

15 identifying a deviation of a position of the second vertical 

line shown on the fourth image with respect to a position of 
the first vertical line shown on the third image; and 

calculating an infinite distance corresponding point 
corresponding to each position on the first vertical line shown 

20 on the third image on the basis of the deviation of the position 
of the second vertical line, 

wherein said step of correcting the location of the search 
range uses the infinite distance corresponding point thus 
calculated as the infinite distance corresponding point 

25 corresponding to the horizontal position of the reference pixel 
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region . 



22. A method of calculating an infinite distance 
corresponding point, the method comprising: 

taking picture of a screen by a pair of stereo cameras 
and producing first and second images of the screen, said screen 
being drawn a first vertical line and a second vertical line 
at an interval equal to a baseline of the stereo cameras; 

identifying a deviation of a position of the second vertical 
line shown on the second image with respect to a position of 
the first vertical line shown on the first image; and 

calculating an infinite distance corresponding point 
corresponding to each position on the first vertical line shown 
on the first image on the basis of the deviation of the position 
of the second vertical line. 

23. A stereo matching apparatus comprising: 

a stereo camera system for taking pictures of a 
predetermined area and producing first and second images of 
the predetermined area; 

a reference pixel region identifying unit for identifying 
a reference pixel region in the first image; 

'a search range identifying unit for identifying a search 
range for the reference pixel region in the second image; 

a correction unit for correcting a location of the search 



range for the reference pixel region based on the amount of 
deviation of an infinite distance corresponding point with 
respect to a horizontal position of the reference pixel region; 
and 

a correlated destination identifying unit for identifying 
a correlated destination of the reference pixel region by 
searching through the search range. 
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ABSTRACT 

The reference image data corresponding to one pixel block 
stored in a reference image line memory 6 is output to a stereo 
matching circuit 8. The comparative image data corresponding 
to a "search range" stored in a comparative image line memory 
7 is output to the stereo matching circuit 8 according to an 
instruction of a search range address generator 13 . The stereo 
matching circuit 8 calculates parallaxes by performing the 
stereo matching based on the reference image data and the 
comparative image data. A position where the search range S 
is set is corrected based on the amount of the deviation of 
an infinite distance corresponding point with respect to a 
position of the reference image data. 



38 



1/5 



UJ o 

o5 



LU 






GE 






RY 




< 


RY 


LU 


MO 




VE 


MO 


O 


LU 






LU 


■z. 










LU 

CC 
UJ 


INE 




PAR 


UJ 

z 


U_ 
LU 


i 






1 


CC 






CO 





C3 



-go 




2/5 



FIG. 2 



WHEN PS = 0 

REFERENCE IMAGE 
LINE MEMORY 6 

COMPARATIVE IMAGE" 
LINE MEMORY 7 



REFERENCE PIXEL BLOCK PB 



SEARCH RANGE S 



A + 0 t 
INFINITE DISTANCE 
CORRESPONDING POINT X 



FIG. 3 



WHEN PS = + 4 

REFERENCE IMAGE 
LINE MEMORY 6 

COMPARATIVE IMAGE" 
LINE MEMORY 7 



^REFERENCE PIXEL BLOCK PB 



SEARCH RANGE S 



A + 4 1 
INFINITE DISTANCE 
CORRESPONDING POINT X 



A +135 



FIG. 4 



WHEN PS = -4 

REFERENCE IMAGE 
LINE MEMORY 6 

COMPARATIVE IMAGE" 
LINE MEMORY 7 



r REFERENCE PIXEL BLOCK PB 



SEARCH RANGE S 



A +127 



INFINITE DISTANCE 
CORRESPONDING POINT X 



3/5 

FIG. 5 



VERTICAL LINES AND NUMBERS 



X REF.NO. 1 REF.NO.2 REF. NO. 3 REF.NO.4 REF. NO. 

COMP. NO. 1 COMP. NO. 2 COMP. NO. 3 COMP. NO. 4 COMP. NO. 5 




2b 



2a 



FIG. 6 



4/5 



COMPARATIVE IMAGE 



COMP. COMP. COMP. COMP. COMP. 
NO. 1 NO. 2 NO. 3 NO. 4 NO. 5 



REFERENCE IMAGE 



REF. REF. 
NO. 1 NO. 2 



REF. REF. REF. 
NO. 3 NO. 4 NO. 5 



FIG. 7 

COMPARATIVE IMAGE REFERENCE IMAGE 



COMP. COMP. COMP. COMP. COMP. REF. REF. REF. REF. REF. 
NO. 1 NO. 2 NO. 3 NO. 4 NO. 5 NO. 1 NO. 2 NO. 3 NO. 4 NO. 5 



FIG. 8 



5/5 



COMPARATIVE IMAGE 



COMP. COMP. COMP. COMP. COMP, 
NO. 1 NO. 2 NO. 3 NO. 4 NO. 5 



REFERENCE IMAGE 



REF. 
NO. 1 



REF. 
NO. 2 



REF. REF. 
NO. 3 NO. 4 



REF. 
NO. 5 




i COORDINATE OF REFERENCE IMAGE 



Declaration and Power of Attorney 
United States Patent Application 



UNITED STATES (Form BDWY-1) 
Patents and Design Patens 
Sole & Joint inventors 



PCT&Noo-PCT 
This form cannot be amended, altered 
or changed after it is signed. 
(For use oniyfor inventors who 
understand the English language.) 



As a below named inventor, I hereby declare that: 

My residence, post office address and citizenship are as stated below next to my name. 

I believe I am the original first and sole inventor (if only one name is listed below) or an original, first and joint 
£T1E£?«S? '^^^'^^^^^^^^^ <>F CALCULATING AN 
INFINITE DISTANCE CORRESPONDING POINT 



(check o, 



m is attached hereto, 
o was filed as U.S. Application No._ 
was amended on _ 



_ and (if applicable) 



□ was filed as PCT International Application No on 

and (if applicable) was amended tinder PCT Article 19 on ___ • 

(I authorize any attorney appointed below to insert information in the preceding blanks.) 

1 have reviewed and understand the contents of the above-identified specification, including the claims, as amended by any amendment referred to above. 
I acknowledge the duty to disclose information which is material to patentability as defined in Title 37, Code of Federal Regulations, §1.56. 

I hereby claim foreign priority benefits under Title 35, United States Code, §119(»H<0 or §365(b) of any foreign and PCT applications) for patent or inventor's 
certificate or §365(a) of any PCT international application which designated at least one country other than the United States of America listed in this 
Declaration. 1 have also identified below any foreign application for patent or inventor's certificate or PCT international application having a filing date before 



Foreign/PCT Application No. 


Country 


Filing Date 


Priority Claimed? (yes/no) 


P. Hei. 11-269182 


Japan 


September 22, 1999 


yes 



1 hereby claim the benefit under Title 35. United States Code, §120 or §365(c) of any United States application and PCT international application designating the 
United States of America listed in this Declaration and. insofar as the subject matter of each of the claims of this application is not disclosed in the pnor United 
States application or PCT international application in the manner provided by the first paragraph of Title 35, United States Code. §112, 1 acknowledge the duty to 
disclose information which is material to patentability as defined in Title 37, Code of Federal Regulations, §1.56 which became available between the filing date 
of the prior application and the national or PCT international filing date of this appli 



U.S. Application No. 


Filing Date 


Status (patented/pending/abandoned?) 








I hereby claim priority benefits under Title 35 United States Code §1 19(e) of any U.S. provisional application^) listed below: 


U.S. Provisional Application No. 


Filing Date 









I hereby appoint the following attorneys to prosecute this application and to transact all business in the Patent and Trademark Office connected therewith: Joseph 
A DeGrandi (17446). Robert G. Weilacher (20531). Richard G. Young (20628). Michael A. Makuch (32263), Dennis C. Rodgcrs (32936), Thomas L. Evans 
(35805), Frank C. Cimino, Jr. (39945), Carolyn Favorite (39183), George A. Metzenthin (P41995), and Steven W. Collier (P42429). 

Send all correspondence to Beveridge, DeGrandi, Weilacher & Young, L.L.P., Suite 800, 1850 M Street, N.W., Washington, D.C. 20036. Facsimiles may be 
sent to (202) 659-1462. Direct all telephone calls to (202) 659-2811. 

I hereby declare that all statements made herein of my own knowledge are true and that all statements made on information and belief are believed to 

be true; and further that these statements were made with the knowledge that willful false statements and tie iike so made are punishable by fine or imprisonmerit, 

or both', under Section 1001 of Title 18 of the United States Code and that such willful false statements may jeopardize the validity of 

the application or any patent issued thereon. 

Full name of sole or first inventor: Yoshiyuki SOGAWA Citizenship: Japan 

Residence (city, state, country): To k yo , Japan 

Post office address: c/o SUBARU Laboratory, 9-6, Osawa 3-chome, Mitaka-shi, Tokyo, Japan 

^- Date: September 1, 2000 



Signature: 

Full name of second joint inventor, if any: 
Residence (city, state, country): 
Post office address: 



Citizenship: 



Signature: 



O Additional inventors and/or prior applications are listed in attached Supplemental Sheets). 



BDWY 39S 



